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(54) Image colour correction based on image pattern recognition, the image pattern including a 
reference colour 



(57) The present invention relates to a method for 
correcting at least one colour of a photographic image 
including at least one pattern area or image pattern with 
a predictably known colour or memory colour, said im- 
age being transferred to a digital representation, where- 
in the method comprises the following steps: said at 
least one pattern area or image pattern is being detected 
with respect to its presence and its location, and prefer- 



ably also with respect to its dimensions; an existing col- 
our in the at least one detected pattern area or image 
pattern is being determined; at least one replacement 
colour value (memory colour) is being provided, said 
value being related to the respective at least one pattern 
area or image pattern and the determined existing col* 
our is replaced by said at least one replacement colour 
value, to correct the colour in the image pattern or image 
area. 
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[0001] This invention relates to a method lor correcting colours ol a photographic image, including at least one pattern 
area and most preferabV a lace image with a orodiciably known colour, wherein the image is in a digital representation 
in accordance with claim 1 Furthermore, the invention relates to a program in accordance with claim 18 a computer 
storage medium in accordance with claim 19. an .mage processing device in accordance with claim 20, a computer 
data signal in accordance with claim 21 and. finally to a data carrier device in accordance with claim 22 
[0002] Photographic .mages are recorcod by means ot photographic image recording devices like cameras (still 
cameras, movedpicture cameras, video cameras d,g.tai cameras, lilmcameras, etc.). The picture data of photographic 
information carried by light is captured by the cameras and recorded, e.g., by means ol a semiconductor memory or 
photochemical on a photographic film. The analogue recorded image information is then digitalised e g by means of 
an analogue-digital (a/d-)converter or by scann.ng a him ,n order to achieve digital image data. The digital image data 
are then processed m order to transform the data to a status in which they are suitable for being displayed for a user 
by means of an output device (e g. printer plus print mod um or screen). 

[0003] Starting from the situation ol reccing ol :n c photography image up to the final display of the image for the 
user or the storage of the image data for a later display thc.c are a lot ol possible sources of error, which may affect 
the photographic image data such that the photograph m^gc displayed to the user is different from the actual ap- 
pearance of the photographic object in particular writ, respect to the recorded colours if compared with the actual natural 
colours. The present invention relates to such coiojr uovi*irjns, 

[0004] The origins foi such kinds of er.ots o. dev„,tu,n s n.* y ta> of a technical nature or may have their origin in the 
way how human beings perceive colours and images Tcchncal causes may be, for instance, chromatic aberration of 
the lens system, colour balance algorithms and d.grtai ca-neras spectral sensitivity of CCD chips or film and in par- 
ticular the application of insufficient colour correct.on aigorrthms The colours of a photographic object captured by a 
camera, of course, depend on the illumination spectrum Contrary to this, the human colour correction system has a 
so-called "colour constancy- feature. The natural humar bong « able to identify colour samples of different colour 
values even under different illumination conditions basoc on h,s memory about the colour value (see "Measurement 
of Colour Constancy by Colour Memory Matching'. Opical Rev«w Vol. 5. No. 1 (1998), 59-63 respectively htte// 
wwwJSSTORJP/OSJ-AP/OpticalReview/TOC-tets^olDVSaO^x htm. The colour constancy is a perceptual.^ 
an.sm. which provides humans with colour vision, which .s -eui-voty independent of suspectorcontent of the illumination 
of a lightsource. Contrary to this, thecolourvalue recordec by camcrasonly depends on the spectrum of the illumination 
light (e.g. tungsten light, flash light, sun light). 

[0005] Additionally, the human being has a good memory lor colours which he often encounters in daily life like the 
colour of skin, foliage, blue sky, neutral or grey (e g me colour of streets is grey). For instance, in the CMYK (cyan 
magenta, yellow, and black) colour space the relationsh.p lor a Caucasian (European) skin tone is 13C-40M-45Y-0K 
This applies at least for young women and children Typcaiiy magonta and yellow is close to equal and cyan is about 
1/3 to 1/5 below magenta and yellow. If magenta is higher man yellow, the skin tone will look red. If yellow is much 
higher than magenta, the skin tone will look yellow. Black should be only in shadow areas of the skin tone or on darker 
skin tones (see. for instance. http:llwww. colorbalance com/html/mcmory html) . 

[0006] Since these kinds of memory colours exist in photography images, they represent characteristic colours for 
photographic images and may be used as a reference Ic colour correction. 

[0007] On the other hand, it is realty difficult, and memory space as well as computer operation is time consuming 
to search through the digital representation of any image :o l.n<j out some reference colours to be able to correct all of 
the colour data ot this image. 

[0008] In the field of automatic detection of particular ,magc patterns it has always been a challenging task to identify 
a searched image pattern in a picture, said image pattern .nc.udmg a memory colour. Such automatic detection is 
recommendable if image data have to be modified or aucrca tor instance to correct a defective recording process For 
instance, if flash light photographs have been made. .1 .s vc- y ,,kciy that such flash light photographs include colours 
which deviate from the actual photographed object itself 

[0009] There are further situations which could a cause cobur delect in a photograph, which can be conected 
However, in the following, the description will be concentrated on the automate detection of facial images since the 
recognition of a skin of colours which are memory colours ol a numan being as referred to above. 
[0010] To search skin colour and a human face in a oom.m .mage, it is known to detect a skin colour at first After a 
skin colour has been detected, it is verified whether m the -eg on ol the colour which is deemed to represent skin colour 
and image pattern of a human face is existent. If this process * aH.rmat.ve. the colour in the face is used to conduct 
a memory colour correction. However, this kind of process « rot applyable if the colour defect in the image is such 
that the colours of recorded human skin can no longer be rtenfhed as human skin, e.g., if skin in a human face appears 
green, orange or grey. HH 

[001 1 J It is the object of the invention to provide a colour correction, wh.cn allows using memory colours for a particular 
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image pattern as a reference for the correction of the colour data of a recorded image. In particular, it is an object of 
the invention to correct a colour or colour of an image on the basis of a memory colour of human skin. 
[0012] The above object is solved by the subject matter of the independent claims 1, 20, 21, 22. 23 and 24. The 
dependent claims are directed to advantageous embodiments. 

5 [0013] The advantages according to the present invention can be achieved on the basis of a method for correcting 
at least one colour of a photographic image including at least a pattern area or an image pattern with a predictably 
known colour (memory colour), wherein this image has been transferred to a digital representation. According to this 
method at least one pattern area or image pattern in particular a human face, is detected with respect to its presence 
and its location and. e.g., its at least approximate dimensions. An existing colour in the at least one pattern area or 

10 image pattern is determined and at least one replacement colour value (memory colour) is then related to the respective 
at least one pattern area or image pattern. Tnis replacement colour value, which corresponds to a so called memory 
colour, replaces then the determined existing colour to correct the colour in the image pattern or image area. In ac- 
cordance with the invention, the human memory colour is used to reconstruct or correct the defective colour in an 
image pattern or pattern area for which a numan being has kept in mind a particular colour imagination. According to 

15 the method of the present invention, it is necessary that at least one replacement colour or memory colour is stored 
for each pattern image or pattern area in particular a human face. Accordingly, since it is possible that recorded images 
are searched through to find different kinds of image patterns, for instance faces, streets, green grass or lawn, or the 
like, it is necessary to store at least one replacement colour, i.e. a memory colour of a human being, for each of these 
image patterns. Accordingly, it is also possible to delect several image patterns or pattern areas in a photograph, i.e. 

20 the digital representation of this photograph mid to replace defective colours in these image patterns by means of 
stored replacement colours, i.e. memory colours which a numan being has kept in mind with respect to the respective 
image pattern. 

[0014] According to an advantageous cnbodi-r»cnt it is possible to determine a deviation between the at least one 
replacement colour value and said cxistirg colour determined in the identified and located image pattern or pattern 
25 area. On the basis of the deviation, it is possible to modify oxisting colour values in the detected pattern area or image 
pattern. This means, the colours in tho doioctod ™*go paltorn are not replaced only by one single colour, the replace- 
ment colour or memory colour, but are on y rx>d»ticd by trie deviation. This means, the image pattern will still include 
different colours also after the colour cmrocton which will look more natural. 

[0015] It is also possible to modify or correct all ntsimg colours of the image on the basis of the deviation. 
30 [0016] Furthermore, it is possiolc to dcormmc an avcagc colour value and/or a mean colour value of the colour 
values in the at least one detected image pattern or oatcrn areas and to use this average or mean value as the existing 
colour to conduct all further procedural steps of the colour correction. 

[0017] Of course, it is also possible to use a distribution ol colour values, the distribution or distributions of which is/ 
are related to one or several memory co ou-s reiatec to the respective at least one pattern area or image pattern. 

35 During this step, a matching replacement colour value is assigned to the determined existing colour or colours. 

[001 8] Furthermore, since it is possible nat the cxist^g colour as well as the assigned colour value or memory colour 
includes different contributions with respect to different colour contents, e.g. a particular red-content, a particular green - 
content and a particular blue-content or tnc udes different contributions of a particular colour space, for instance a 
HSV colour space, the contributions having io tc considered in a particular manner, it is possible that a transform is 

-to necessary to modify the colour values ct tr*e cngna digital representation of the original image. Accordingly, by means 
of a matching transform, it is possible to consider al colour contributions with respect to a particular colour to be 
corrected in an appropriate manner 

[0019] A further embodiment is based on tie recognition of one or several particular image patterns, like a human 
face, a street or the like, the image patterns including a particular colour which is memorised by the human being on 

^5 the one hand, and, on the other hand, the image oar.crn can be detected in a digital representation of a recorded image 
in a comparatively short time. Furthermore, the respective image pattern which can comparatively easily be detected, 
like a human face, includes a memorised colour like the colour of the skin of a human being. On the basis of the 
recognition of a particular image pattern and the recognition of a particular colour of this detected image pattern, it is 
possible to correct the colours of a photograpmc image by correcting all colours of the image considering the deviation 

50 between the colour detected in the detected image pattern and the memorised colour, which a human being would 
have expected to perceive in the detected imago pattern, like for instance a face, a street, or the like. 
[0020] According to the invention, it is possible to use any existing methods for image pattern recognition. 
[0021 ] For the actual detection of faces, any system that fulfils this reasonably well will do. This could be for instance 
a neural network approach, as proposed by Henry Rowley. "Neural Network-Based Face Detection", PhD Thesis 

55 CMU-CS-99-117, Carnegie Mellon University. Pittsburgh 1999, or some wavelet-based approach, as proposed by 
Schneiderman et al, "A Statistical Method for 3D Object Detection Applied to Faces and Cars", Proc. CVPR 2000 : Vol. 
I, pp. 746 - 752, Hilton Head Island 2000. Of importance at this stage is that the detection of faces happens fully 
automatically and that the detection rate is reasonably high and the phase negative rate, that is, faces being detected 
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[0030] ,n accordance with the invention, it is therefore posstole to automatically correct the colour of a complete 
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recorded image on the basis of the colour of only one particular image pattern or pattern area, like a face. 
[0031] The colour correction of the present invention allows to calculate and perform the colour correction of a digital 
photographic image in such a way that memory colours are reproduced in an optimal way. The invention can in particular 
be applied to photographic DMD printers, photographic ink jet printers, photographic CRT printers, photographic lab- 
5 oratories, in particular photographic compact laboratories, also called "minilab". 

[0032] Those printers or laboratories process received photographic image information. The photographic image 
information may be received classically on films or may be received digitally via networks (e.g. Internet. LAN, etc.) or 
via storage media (CDROM, disks, memory chips, etc.). 

[0033] The colours used as a reference for the colour correction according to the present invention are called "ref- 
10 erence colours". Those reference colours typically correspond to memory colours and represent colours characteristic 
for a significant part of most photographic images. Therefore, those kinds of characteristic colours (memory colours) 
may be derived from a plurality of photographic images, which may be selected e.g. statistically or by photographic 
experts. Based on this plurality of photographic images, a model for the characteristic colours (memory colours) may 
be derived, which provides the colour values which the characteristic colours (memory colours) usually should have. 
15 These colour values can be used in the shape of colour value distributions, representing likelihood's for a certain colour 
value. 

[0034] The inventor of the present invention has considered that a memory colour is not represented by just one 
exact colour value, in reality, but by a plurality of colour values. According to the present invention, this plurality of 
colour values representing a particular memory colour (characteristic colour) may be described by means of at least 

20 one distribution, which describes the distribution or distributions of colour values in a colour space. The distribution 
describes, in particular, a two or three-dimensional range or section in the colour space. The distribution may not only 
relate to a colour value : i.e. its position in colour space, but may also relate to one or more parameters of the colour 
values described by the distribution. For instance, a parameter may relate toaprobabilitythatacolourvalue represents 
a particular memory colour. This probability may, for instance, be deduced from the statistical abundance of the colour 

25 value in a plurality of photographic images. In this preferred case, the distribution represents a probability distribution. 
According to another example, a parameter may represent a weighting factor for the correction procedure, i.e. a meas- 
ure for the importance of the colour value for the representation of a memory colour. Usually,, the colour values are 
more important the higher the abundance or the higher the probability is. 

[0035] Additionally several different distributions may be provided for one and the same memory colour in case 
30 additional information about the image capture situation is available. If, for instance, the digital camera stores that the 
image has been taken under flash light conditions, a distribution adapted to flash light conditions or based on a plurality 
of flash light photographic images may be used instead of a standard distribution, which covers all kinds of image 
capture situations (sunlight, flash light, in-house). However, preferably, this kind of additional information is used to 
determine the so-called prior knowledge as described below and, thus, if no additional information is available, pref- 
35 erably only one distribution is assigned to one and the same memory colour. According to the present invention, the 
memory colour is used as a reference colour. Preferably, a set of reference colours and, thus, their corresponding 
distributions is provided. The predetermined data on the distributions may be stored in a memory unit and/or may be 
accessed via network on demand and may be updated, e.g. based on new statistical data. 

[0036] The colour correction method or the colour correction device of the present invention receives the image data, 
40 which are to be corrected, and which represent a photographic image. The image data are preferably received in digital 
form, e.g. via a storage medium or via a network. Alternatively or additionally, the colour correction device of the present 
invention may comprise a scanner, which scans a photographic film in order to produce the digital photographic image 
data. 

[0037] The colour values of a recorded image are usually digitalised and may tor instance, be represented by a 
45 three-dimensional vector, the components of which has integral numbers (e.g. 0 ... 255). Different colour spaces may 
be used to describe the colour values, e.g. RGB, sRGB, CMYK, Lab, CIELab, etc.) to obtain a digital representation 
of the image. 

[0038] According to the invention, a reference colour and/or the corresponding distribution (or selected distribution) 
is assigned to the identified and located pattern area or image pattern. The assigned distribution is selected out of the 

so set of available distributions. 

[0039] Based on the distributions assigned to the image pattern or, in other words, based on the reference colours 
(memory colours) assigned to the image pattcrn(s) of the imago, a transformation is determined. The transform rep- 
resents a manipulation of the image data for correction purposes. The transform is determined based on the colour 
value or colour values present in the one or more of the image patterns. These colour values represent the starting 

55 point for the transform. The distributions define the end point for the transformation to be determined. The aim is that 
the colour values, of the image pattern match the colour values described by the distributions and which a human 
observer would expect to see. Based on the determined transformation, the colour values of the image data, preferably 
of all image data may be transformed in order to achieve a corrected image. The basis for this correction are the 
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[0048] If it is already known that the corrected image data will be passed to a particular output channel (e.g. a printer 
or minilab) and if the colour management profile (such as an ICC profile; International Colour Consortium, http://www. 
color.org) is known, then this knowledge can be used during the step of determining the transformation, in particular 
. during the corresponding optimisation process. For this purpose, the determination of the transformation is performed 
5 such that the transformation comprises a colour management transformation, which corresponds to the colour man- 
agement profile of the output channel. 

[0049] Additionally or alternatively, the correction may be performed in view of the human colour perception of the 
image. For this purpose, a colour appearance model (such as CIECAM97s, Mark Fairchild, "Colour Appearance Mod- 
eling and CIECAM97s". Tutorial Notes (CIC99), 1999. location: Armin Kndig ) may be used. The colour appearance 
10 model may be represented by a transformation, i.e. a colour appearance transformation. The transformation used for 
correction according to the present application is then determined such that the transformation comprises such a colour 
appearance transformation. 

[0050] The present invention is not only directed to a method, but also to a program and a computer storage medium 
comprising the program; Additionally, the present invention is directed to a photographic image processing device, 

J5 which performs the above-described correction processes. Such a photographic image processing device preferably 
comprises a memory unit, which stores the distributions, an input unit, which receives the digital image data, a selecting 
unit, which selects the reference parts, an assignment unit, which assigns the distributions to the reference parts, a 
determining unit, which determines the transformation by considering the above discussed matching, and a transform- 
ing unit, which performs the correction transformation. Such a photographic image processing device may be imple- 

20 mented by ASICs, hardwired electronic components and/or computers or chips programmed in accordance with the 
method. Furthermore, the invention relates to a photographic printer or photographic laboratory, in particular a photo- 
graphic minilab, which performs the method described above, which comprises the above described photographic 
image processing device. Each device may comprise a data processing device, e.g. a computer, on which the above- 
mentioned program runs or is loaded. 

25 [0051] In the following, the invention will be described in more detail with reference to embodiments in accordance 
with the present invention. These embodiments are described with reference to the accompanying drawings and dis- 
close further advantageous aims and features in accordance with the invention. In the drawings: 

Fig. 1 shows a flow diagram for face detection in a refined version. 

30 Figs. 2 and 3 depict face pictograms to be identified in a digital representation of an image. 

Fig. 4 shows memory colour models for "neutral" (full line), "blue sky" (dashed), "skin" (dotted), and "foliage" 

(dash-dotted). 

Fig. 5 shows prior knowledge distributions p (log(rf), log(gf)) for digital cameras in general (top) and for a 

particular model (Kodak DC 210 zoom, bottom). 
35 Fig. 6a • shows an optimisation via forward modelling, in accordance with a basic embodiment of the present 

invention. 

Fig. 6b shows an optimisation via forward modelling, where the basic embodiment is combined with colour 

management for a known output channel. 
Fig. 7 shows a schematic structure of a photographic image processing device, which may also be called a 

40 colour correction device in accordance with an embodiment of the present invention. 

[0052] In the following, the principles of the method of the present invention will be discussed with reference to the 
detection of a human face and with respect to the detection of skin in this face. Of course, also other image patterns 
can be searched in which other memory colours can occur. 

45 [0053] For the actual detection of faces, any system that fulfils this reasonably well will do. This could be for instance 
a neural network approach, as proposed by Henry Rowley, "Neural Network-Based Face Detection", PhD Thesis 
CMU-CS-99-117, Carnegie Mellon University, Pittsburgh 1999, or some wavelel based approach, as proposed by 
Schneiderman et at, "A Statistical Method for 3D Object Detection Applied to Faces and Cars", Proc. CVPR 2000 : Vol. 
I, pp. 746 - 752, Hilton Head Island 2000. Of importance at this stage is that the detection of faces happens fully 

so automatically and that the detection rate is reasonably high and the false negative rate, that is, faces being detected 
even though there is no face present, is reasonably low. What reasonable constitutes will depend on the actual context 
of the application. The disclosure of the Rowley and the Schneiderman references is incorporated into this application. 
[0054] As most face detectors are not invariant to rotation, it can be useful to ensure that all the possible orientations 
of faces can be detected. How to do this will highly depend on the face detector being used, as the rotation invariance 

55 of each detector will vary widely. For instance, in Rowley's approach, rotation invariance is given within approximately 
± 15°. On the other hand, in the approach by Schneiderman, rotation invariance is given in a range of about ± 45°. 
Therefore, rotation invariance has to be ensured by external means, this can for instance be done by pre-rotation of 
the image, followed by a post-processing and the normal face detection. This is shown in Fig. 1 . 
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[0067] ^eabovegivenrepresentationofthecolourvaluesasrgbvaluesisonlyanexampleandotherrepresentation 
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of the colour value, e.g. by means of Lab vectors, may be chosen. The transformation T transforms the rgb values into 
the new pixel values r'g'b'. This transformation can be as complicated as is necessary to be appropriately applicable 
in accordance with the invention. Examples for transformations are disclosed in G. Wyszecki and W. Stiles, Colour 
Science: "Concepts and Methods, Quantitative Data and Formulae", Wiley, 1 982. For instance, the transformation may 
5 be as follows: 

• The rgb values are simply scaled. This kind of correction is often done in digital cameras. In this case, the trans- 
formation T corresponds to a diagonal matrix in which the components of the matrix correspond to multiplication 
factors. 

10 • The colour values may be transformed from one colour space into another colour space by the transformation. 

For instance, the rgb values may be transformed to colourimetric XY2 values and then these values are scaled. 
Preferably, the colour values of the image pattern are transformed into a colour space in which one dimension 
represents the luminance or lightness and the other dimensions, independent therefrom, describe the colour hue 
and the colour tone. 

15 • The transformation may transform rgb values or any other kind of colour values into LMS Cone response values 
and then these values are scaled. 

• The transformation may represent the application of a general 3x3 matrix in any of the above-mentioned colour 
spaces. The matrix may represent a rotation, deformation, or displacement in colour space. In particular, if one of 
Ihe dimensions of the colour space represents luminance or brightness, the transformation may be constructed 

20 such that the luminance value is kept constant. For instance, the transformation may comprise a matrix, which 

describes a rotation around the luminance or brightness axis. 

[0068] A model for memory colours, which relates to distributions of colour values corresponding to the memory 
colours, is a probabilistic model. Each memory colour A k (A, = neutral or gray, A 2 = blue sky, A 3 = skin, A 4 = foliage) 
25 is defined via its likelihood: 

p(a,blA k ). (2) 

30 [0069] The above expression describes the probability that a colour value represented by the parameters a and b 
belongs to the memory colour A k . Only as an example, it is assumed in the following that the parameters a and b 
correspond to the components a and b of the Lab vector. The above expression represents a conditional probability 
and describes the probability of a colour value a, b under the condition of a memory colour fi^. 
[0070] The detailed shape of the above equation (2) can be as complicated as necessary to describe the training 

35 data, e.g. to describe the result of a statistical analysis of memory colours in a plurality of photographic images. The 
inventors have achieved satisfying results, when they describe the probability distributions with two-dimensional, mul- 
tivariate Gaussians. The Fig. 4 depicts examples for memory colour models (probability distributions) of "neutral" (full 
line), "blue sky" (dashed), "skin" (dotted), and "foliage" (dash-dotted). The probability distributions are shown such that 
the Gaussians are depicted at 50 % maximum probability of each memory colour, i.e. p(a, b I A 1 ) = 0.5 for all colour 

40 values, which have an (a, b) value which lies on the full line in Fig. 4. 

[0071] In the following, it is described in which way the transformation is determined in order to achieve the best 
matching between the transformed colour values of the image patterns and the colour values of the probability distri- 
butions of the replacement colours assigned to the image patterns. The method described in the following is an opti- 
misation method or algorithm. 

45 [0072] The transformation T is characterized by a certain number of parameters (e.g. the scaling factors rf , gf, bf) 
representing the diagonal components of a 3 x 3 matrix. These parameters are determined from the input colour values 
r h g h bj of the identified and located image patterns i in such a way that the transformed pixels r' h g'j, b'j correspond to 
the optimised realisation of the corresponding memory colour Aj as good as possible, given the image patterns and 
Ihe colour values of the image pallern. 

so [0073] The degree of "as good as" may be defined in the a-b colour plane of the Lab colour space. The components 
of the Lab colour space may also be designated as L*. a*, b* (see, for instance, Fig. 4). In this case, the components 
relate to ClELab. Psychological studies (K. Toepfer and R. Cookingham , 'The Quantitative Aspects of Colour Rendering 
for Memory Colours", in 1ST PICS2000 Conference, pages 94-98, 2000, location: MS) shows that this Lab colour space 
is well suited to define memory colours and thus to define replacement colours. 

55 [0074] Given a particular transformation T e (0 denotes the parameters of this transformation), we can calculate the 
a and b values of the image patterns i as . 
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■ ft, i" f «( r '|.0 , |.b , |)-f a re(r, l g, l b i )) (3) 
b, i = f b(rVg , pb' i )=f b fTe(r j ,g i :b l )) (4) 
SoTe'RGB? ^ d9n0te ^ fUnCti ° nS " Ca,CU,3te ^ 3 and b <™ «* — -.our space of rgo (e.g. sRGB or 



P(D|G) = n p(a': ) b' i |m) 



(5) 



[0077] Based on Bayes' equation, the posterior conditional probability may be obtained: 



P(0 I D) a p(D I 6) • p(fi) 



(numerical technique) for ZShTSXI wS a nl T d6SCenr repreSSntS an 'P^on technique 

known, an upper limit for the amount of ejection by £ ^^S^SSST a ' 9 ° rithmS * 

be deduced, based on which p(6) may be determined ' ° d ° f ' nVention ^ 

• ^""'"-'^aybebasedonaddit™^ 
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the image may be classified into a class. The images, which are members of a particular class, have a particular 
systematic bias in their colour appearance, which may be used to determine p(6). For instance, the Images may 
be classified in sunset images, portrait images and so on. 

5 [0081] Mathematically speaking, prior knowledge of the colour correction is always available as probability distribu- 
tion 

P<«) (7) 

10 

and can be included in the process of inference via ccuation (6). 

[0082] The colour correction method of the present invention can preferably be combined with a colour management 
method or the colour correction device of the present invention comprising preferably a colour management unit. The 
procedure of optimisation of the transformation T described aoove is. in principle, an optimisation using a forward 

15 model, i.e. the colour transformation T is changed jnw the modified (transformed) colour values optimally match the 
models of ideal memory colours, i.e. the colour values d the colour distributions corresponding to the replacement 
colours. In the basic workflow, this match is done in a standardised colour space (e.g. a*b* plane of L*a*b*); However, 
if it is already known that later the image will be passed to h particular output channel (e.g. a minilab) with a known 
colour management profile (such as an ICC profile international Colour Consortium, http://www.color.org) then this 

20 knowledge is preferably used during the optimisation process 

[0083] The colour profile relates to the colour values of the input data which are input into the output channel to the 
colour values which are output by the output channel (output device). Assuming, for instance, that the image data input 
in the output channel express the colour values as rgb values and that the colour values expressed by the output signal 
of the output channel arc represented as Lab values then the colour profile contains the information of which Lab 

25 values are to be expected on the output for which mpui rqb valuos Tho Lab values relate, for example, to those Lab 
values, which are measured when optically analysing the pnntout of a printer, which represents the output channel. 
This optimisation step can be done in such a way as to optimise the reproduction of memory colours output by the 
output channel (e.g. the memory colours on the printout) 

[0084] The Fig. 6a shows the basic optimisation loop The data rgb are input in the colour correction process of the 
30 present invention and are to be corrected by a correction transformation T. As a result of the correction transformation 
T colour values r'g'b' are obtained. These colour values arc subjected to a colour space conversion in order to obtain 
L*a*b* colour values. Based on the memory colour model (colour distributions), which represent the information on the 
replacement colours or ideal a*b* values, it is chocked in a step in accordance with Fig. 6a whether the a*b* values 
obtained after the colour space conversion match with the ideal a*b # values. The colour correction transformation T is 
35 changed until the matching is optimised. This may be done fo f instance, iteratively as indicated by the optimisation 
loop in Fig. 7. 

[0085] The colour correction transformation T and the colour space conversion may be represented by a transfor- 
mation T, which comprises both the colour correction transformation T and the colour space conversion. The optimi- 
sation loop is then performed in order to optimise the (overall) transformation T. 
40 [0086] The Fig 6a depicts the optimisation via forward noddling The basic optimisation procedure of Fig. 6 is 
combined with colour management for a known output channel The overall transformation T comprises instead of the 
colour space conversion transformation a colour management transformation. Of course, according to an alternative 
embodiment, the overall transformation T may compnsc Dotr. a colour management transformation and a colour space 
transformation. Furthermore the sequence of the correction transformation T and the colour management transforme- 
rs tion or the colour space transformation may be changed ic the colour space transformation or the colour management 
transformation may be performed before the colour correction transformation. 

[0087] The colour management transformation corresponds to an application of a colour profile on the r'g'b* colour 
values in order to achieve output values, which are expected to be output by the output channel (output device). If, for 
instance, the output device is a printer, the colour management transformation results in L*a*b* colour values, which 

50 are expected on the prints produced by the printer. As in Tig 6a the quality of the matching between the transformed 
colour values and the colour values is checked, which results from the memory colour model (ideal a*b*). 
[0088] Additionally or alternatively to the colour managemert transformation a colour appearance transformation 
may be incorporated in the optimisation loop shown in Fig 6a nnd in Fig 6b. If this is the case : the overall transformation 
V comprises not only the correction transformation T but at least also a colour appearance transformation. The colour 

55 appearance transformation represents a colour appearance mode? If the colour appearance transformation replaces 
the colour management transformation in Fig. 7b, this would mean that neither the theoretical colour (basic optimisation) 
nor the paper colour (basic optimisation plus colour management model) but instead the perceived colour is optimised 
using MCPCC. 
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r0 °fu T!f ^.t e 6aSily d ° ne by subs,ituti "9 the colour management engine in Figs. 6 by a colour appearance 
19M l^tto^ a l^rf 7 ^' d Fairchild - ° Co,our Appearance Modeling and CIECA^97s\ Tutoria°NotK (aC99) 
ISSS? t"T ° 0l0Ur a PP earance ""storm, w*'h represents the colour appearance 

under which a human being perceives the colours. For instance, the colour values may be adjusted toa typical illumi 

colTr TJX " h 6 , 5) ' 3 ? Ca ' baCk9r ° Und C0 ' 0Ur °" WhiCH *° imaae is l00ted at «* ^ thet Ind 

HiElSSf 7 "7, f? ^ in,,UenCe °" CO ' 0Ur PerCep,i0n ' e - 9 - the medium ma V be shi "V (brilliant) or mat 
S !^, " IUm ' na,i0n (bri 9 htness > ma Y have a " on the perception* the colours by 

SToks at the !mage ^ ** inS,anC6, ,0 i,IUmina,i ° n S ' rength ' When 3 human 

[0090] It has to be kept in mind that the colour correction according to this invention is accomplished by detecting at 
eastone,^ 

which 

m aZSancZS 0 " " "»* ° f inVemi ° n " 01 3 C0 '° Ur C0 " e * i0n device ^ ^es 

or IZS Tl °"- reCeiVin9 Unrt 100 receiV6S the ima 9 e da,a > which "»* '<* in stance be a modem 

or a network part. The receiv.ng unit passes the image data to the selecting unit. The selecting unit may for instance 

3TJLT" ln ! Uni, H WhiCh a " 0WS Se,eCt,n9 31 ' eaSt ° ne ima9S Pa " em - The ^ aZs 

from the selecting un,t to the assignment unit. The assignmen. unit accesses the provisioning unit, which may be a 

Z ZZorT™™ J T? mem0ry C °' 0UrS ' 0r,he """P™** Pa«-n S or the coZdSr 
J!!" mem °7 CO ' ours ,0 ,ne assignment unit upon request. The assignment unit assigns the appropriate 
memory colours or colour distributions to the corresponding image patterns. The image patterns together with the 

able Th?lf h 3 T dCtCrn,inCS ,ransforma t- •* ^ -°ans of tho optimisation loop described 
TZ tranS,0mi ation is passed to the transforming unit 600. The transforming unit 600 receives the 

"V? ima9e d3ta " aCCOrdanCe With t"e transformation in order to 

tio ZvL f ' Wh ' Ch thSn ° U,PUt by the P h0, °9 ra Phic image processing device or colour correc- 
tion device of the present invention. 

E on l S in! S ^' math ° d 30 ° bjeCt de,eCti ° n Can a ' S0 be USed S,a,is,ics of both j ™ge Pattern appearance 
and non-image pattern" appearance using a product of histograms can be employed. Each histogram represents the 

SET" 3 SUbS6t ° f WaVe ' et C ° effiCientS and th6ir P ° S " i0n on ,he ^Pattern. This approa" iMo use manJ 

KrthliSf T T" apPearanCe is the main P roblem here - For example, faces vary in shape, size, colouring 
ZtlT de,a,ls ^ lsual appearance also depends on the surrounding environment. Light sources will vary in their 

on 1 T ad ?° na ' H9ht ° n ^ ima9e Pattem - The appearance of the ima 9 e P a «ern also depends 
on s pose, that is, its position and orientation with respect to the camera. For example, a side view of a human face 
w.l look much drfferent than a frontal view. An image pattern detector much accommodate all this variation and s^ 
distinguish the image pattern from any other pattern that may occur in the visual words 

E„ nl h c! ref ° re ' 3n image P attern detection witn tW0 sta 9 es *° r ima ge pattem detection is used. To cope with var- 
of he ? V ,' eW " baSed aPPr ° aCh ^ mUltiP ' e de,eCt ° rS mat are each s P ecia '^d to a specif ic orientation 

vlriation P ***** m ° de " in9 ^ °' ^ det6Ct ° rS iS accomplished to account for the remaining 

fn„°! 5 L 0 S H P ! Ci f $6d d ? Ct0rB arS US6d 6aCh 01 th6m ^P'" 9 Wi,h a ■»* cMc orientation of the image pattern Accord- 
ing, one detector may be specialised to left or right profile views of faces and one may be specia.ised to frontal vTews 

lased d T"t ^ aPP ' ied " Para " el lh6ir feSU,lS are ,h a" combined. For human ,aces"woTw 

SSJSrS^ t ^ ' 16 eXamP ' e fr ° ntal and ri9ht Pr0,ile ' T ° d6teCt ,eft "P rofile ^s. » is possible to direct 
the right prof^e detector to mirror reversed input images. Each of the detectors can not only be specialised in orientation 

butcar ia.se be designed to find the image pattern only at a specified size within a rectangular image wind o rT~e 
to be able to detect the image pattern or face a, any position in an image. The detectors will be re-appliedTaCsS 

Z£l °L I"'' hT' 3 : Wind ° W - Then 10 bG abl ° t0 dCtCCt thC ima9C P a «cm at any size tho input mago3be 
n„o C1 T u . V and thS deteC, ° rS Wi " bS re - applied in the same fashion to each resized image 
n ,k!, ,„ deteCt0rS US6S tHe Same under| y in 9 f orm for the statistical decision rule. The detectors differ only 

in that they use statistics collected from different sets of images V 

[0097] There are two statistical distribution which can be modelled for each view-based detector The statistics of 

oa^r 9 ;, ^T' h P(ima9e ' ° bjeC,) and S,a,iS,iCS 01 thS r6St ° f ,he Visual whi ch we cal.^e ^o mage 
pattem class P(,magel object) are modelled. Then a detection decision will be determined using the likelihood ratio tS 
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P {image | object) A _ P(noE-object) A 

P(imflge|Don-object) ' I" P( object) y 0) 

5 . 

[0098] If the likelihood ratio (the left side) is larger than the right side, we decide the image pattern is present. 
[0099] The likelihood ratio test is equivalent to Bayes decision rule (MAP decision rule) and will be optimal if the 
representations for P(image! object) and P(imagel non-object) are accurate. The rest of this section focuses on the 
10 functional forms being chosen for these distributions. 

[0100] In the equations, the term image pattern or pattern area is represented by the term object and non-object, 
respectively. 

[0101] The difficulty in modelling P(image I object) and P(image I non-object) is that the true statistical characteristics 
of appearance either for the image pattern or for the rest of the world are not known. For example, it is not known 
'5 whether the true distributions are Gaussian, Poisson, or multimodal. These properties are unknown since it is not 
tractable to analyse the joint statistics of large numbers ot pixels. 

[0102] The approach here is to choose models that are flexible and can accommodate a wide range of structures. 
[0103] Histograms are almost as flexible as memory-based methods but use a more compact representation whereby 
the probabilily is obtained by table look-up. Estimation of a histogram simply involves counting how often each attribute 
20 value occurs in the training data. The resulting estimates are statistically optical. They are unbiased, consistent, and 
satisfy the Cramer-Rao lower bound. 

[0104] The main drawback of a histogram is that only a relatively small number of discrete values can be used to 
describe appearance. To overcome this limitation, multiple histograms are used where each histogram, P k (image I 
object), represents the probability of appearance over some specified visual attribute, pattern^ that is, pattern^ is a 
25 random variable describing some chosen visual characteristic such as low frequency content. The appearance has to 
be partitioned into different visual attributes. However, in order to do this probabilities from different attributes have to 
be combined. 

[0105] To combine probabilities from different attributes, the following product has to be taken where each class- 
conditional probability function has to be approximated as a product of histograms: 

30 

P{image\objcci) ~ Y[^k(P attern k\ °^J ect ) 

35 " (2) 

P {image \ non-object) = k(P attern k jnoc-object) 

k 

40 [0106] In forming these representations for P{image\ object) and P(image\ non-object) it is implicitly assumed that 
the attributes (pattern k ) are statistically independent for both the image pattern or object and the non -image pattern or 
non-object. 

[0107] In choosing how to decompose visual appearance into different attributes the question of what image meas- 
urements to model jointly and what to model independently can be delt with. 
45 [0108] Obviously, if the joint relationship two variables seems to distinguish the object or image pattern from the rest 
of the world, it should be tried to model them jointly. If the results are uncertain, it is still probably better to model them 
independently than not to model one al all. 

[0109] For faces and also for other image patterns it is necessary to jointly model visual information that is localised 
in space, frequency, and orientation. Accordingly, the visual appearance along these dimensions has to be decom- 
so posed. The appearance of the object or pattern area has to be decomposed into parts whereby each visual attribute 
describes a spatially localised region on the object. By doing so the limited modelling power of each histogram will be 
concentrated over a smaller amount of visual information. 

[0110] Since important cues for faces and cars occur at many sized, multiple attributes over a range of scales are 
necessary. Such attributes are to be defined by making a joint decomposition in both space and frequency. Since low 
55 frequencies exist only over large areas and high frequencies can exist over small areas. Attributes with large spatial 
extents are defined to describe low frequencies and attributes with small spatial extents are defined to describe high 
frequencies. The attributes that cover small spatial extents will be able to do so at high resolution. These attributes will 
capture small distinctive areas such as the eyes, nose, and moth on a face. Attributes defined over larger areas at 
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lower resolution will be able to capture other ™ P onan. cues. On a face, the forehead is brighter than the eye sockets 
to hoL 15 T attnbUteS Wi " be decomposod « content. For example, an attribute Ihat \ 

lelSnea^s " Qrea ' er rc ~''°" al •» ™«"» Matures than « ft abo had 

22? ' ^ dec ° mp ° sins tne ^ or ™W P*»crn spatially, it is not intended to discard all relationship 
human ace £ " ^ "■""»'•«» ° f ,hc W * ™ important cue for detection. For ex^mpte on a 

iZ <t n , 6yeS / T m ° U,h ^ fi " XOd 90omo,ric ""figuration. To model these geome^c relation 
TH P0S, "° nS °' 6aCh a ' ,ribu,e samplc w,,h '«***' 10 « »o««"ate frame affixed to the objec?have T be epre- 

ZFJSE"""™?*™ ^ San,p,CS rClf,,,V0 P0Si "° n With respect 10 a » ,he ° th ^- vES! ^ref- 
lation, each histogram now becomes a join, dtstnbuuon of attribute and attribute position PApatternJx v) x vTobf^n 

and P„o»«e«v(x.y). x,y I non-object). whoro attnbute pcs.hon. x.y. is measured with ip^^n^S 

oTsS ,oT ver ? attnbu,e posi,ion is no ' rop,cscmcd at lhe oridinal resoiuto " ^^.TSZTi*: 

va^ns^ 

[01 13] To create v.sual attributes that are localised m space, frequency, and orientation it is necessary to be able to 
easily select .n.ormat.on that ,s localised along these o<mcns,ons It k advantageous to ran fomi h Hma^tto a 

ezzx* ? ese ot r ,rans,omis - h * posst>,c ,o ■» *. iizr: 

the number of Transform coefficients is equal to the ong.n*! number of pixels 

K2oJn".S!!?^^ 0 ^ i,a " ""^ SJbt>nndS * hat toC8liSed ° rientati0n and fre <W 
Wrth.n each subband, each coefficient * spatially locaiscd A wavelet transform based on 3 level decomposition usina 

^ZCT^: 9 n 9 7 Can ? " diSC, ° SOd ° S ' ran9 and T ^ WaV6,etS and Fii^KSZ 
siey-Cambndge Press, 1 997, producing 10 subbands as shown boiow: 
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[0116] Each level in the transform represents a higher octavo ol frequencies A coefficient in level 1 n^m^ a 
L^eno^ 

teir^ 

coefficients ,n level 3 LH band. This attribute would capture n, g n Ucquency horizontal patterns ov I sr£.7ex£m in 
S Z«T a9Q - ,; n ° ,hCr Patt ° rn "* C0UW rCPrCSCm SP " U "* 2* blocks in the LHand HL^ndTof the 

m£T S^l W ° U H reP K eSent " imermediate ' reqUCnCy ban ° ° VOr 3 ,a ^ r spalial extenl - the image. 
[01 18] Since each attribute must only take on a finite number 0 i values a vector quantization of its sampled wavelet 
coefficients wil. have to be computed. To keep histogram we undo. e.g. 1.000.000 JT^SZE!^ 
expressed by no more than e.g. 10,000 discrete values s.nco . y (pos „,L will together ake on a^u^l 00 discrete 
values. To stay within this limit, each visual attribute will be oefned to samp.e 8 wavelet coeffteiente at a time and w H 
quanhze each coefficient to 3 levels. This quantization scheme g.es 3*^.561 discrete •S^J^JST 
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[01 19] Overall, e.g. 1 7 attributes are used that sample the wavelet transform in groups of 8 coefficients in one of the 
following ways. 

[0120] A: Intra-subband - All the coefficients come from the same subband. These visual attributes are the most 
localized in frequency and orientation. 7 of these attributes are defined for the following subbands: level 1LL, level 1 

5 LH, level 1 HL level 2 LH. level 2 HL level 3 LH ? level 3 HL. 

[0121] B: Inter-frequency-Coefficients come from the same orientation but multiple frequency bands. These attributes 
represent visual cues that span a range of frequencies such as edges. 6 such attributes are defined using the following 
subband pairs: level 1 LL - level 1 HL. level 1 LL - level 1 LH. level 1 LH - level 2 LH, level 1 HL - level 2 HL, level 2 
LH - level 3 LH, level 2 HL - level 3 HL 

10 [0122] C: Inter-orientation - Coefficients come from the same frequency band but multiple orientation bands. These- 
attributes can represent cues that have both horizontal and vertical components such as corners. 3 such attributes are 
determined using the following subband pairs: level 1 LH - level 1 HL. level 2 LH - level 2 HL, level 3 LH - level 3 HL. 
[0123] D: Inter-frequency / inter-orientation - This combination is designed to represent cues that span a range of 
frequencies and orientation. One such attribute combining coefficients is defined from the following subbands: level 1 

15 LL, level 1 LH, level 1 HL. level 2 LH. level 2 HL. 

[0124] In terms of spatial-frequency decomposition, attributes that use level 1 coefficients describe large spatial 
extents over a small range of low frequencies Attributes that use level 2 coefficients describe mid-sized spatial extents 
over a mid-range of frequencies, and aitribuics that use level 3 coefficients describe small spatial extents over a large 
range of high frequencies. 

20 [0125] Afterwards each attribute i* bamyiod at regular intervals over the full extent of the object, allowing samples 
to partially overlap. Our philosophy in doing so is to use as much information as possible in making a detection decision. 
For example, salient features such as the eyes aid nose will be very important for face detection, however, other areas 
such as the cheeks and chin will also help but perhaps to a lesser extent. 
[0126] Thus, the final form of tnc detector is given by 

25 
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where "region" is the image wincow (sec Section 2) tc be classified. 

[0127] Now. the actual histograms for P. [p<ittorr\ix y) x y I object and P k (patteri\(x 7 y),x,y I non-object) have to be 
40 developed. In gathering statistics one of the immediate prooicms is to choose training examples for the class "non- 
object" or non-image pattern. Conceptua ry this class represents the visual appearance of everything in the world 
excluding the object to be classified In order to achieve accurate classification it is important to use non-object samples 
that are most likely to be mistaken tor the object T his concept is similar to the way support vector machines, work by 
selecting samples near the decision boundary as disclosed in V.N. Vapnik, The Nature.of Statistical Learning Theory, 
45 Springer, 1 995, . To determined such samples a method called bootstrapping can be used. In bootstrapping , preliminary 
detector can be trained by estimating P k \pattcn\{x y}.x.y I non-object) using randomly drawn samples from a set of 
non-objects images. Then, this preliminary detector is applied lo a set of about 2000 images that do not contain the 
object and select additional samples at those locations that gave high response. 

[0128] We collect P k (patterr\{xy).xy I object) from images of the object. For each face viewpoint about2,000 original 
so images are used. For each original image around 400 synthetic variations are generated by altering background scenery 
and making small changes in aspect ratio orientation, frequency content, and position. 

[0129] Statistics for these training examples can be gathered using several approaches. For the face detector, the 
classification error is minimized over the training set. by using the AdaBoost disclosed in Y. Freund, R.E. Shapire, "A 
Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting", Journal of Computer and Sys- 
55 tern Sciences, 55:1 , pp. 119-139. 1997. and R.E. Shapire. Y. Singer "Improving Boosting Algorithms Using Confidence- 
rated Predictions", Machine Learning 37:3. pp. 297. 336. December, 1999, algorithm. AdaBoost works in an iterative 
fashion. First, a detector is trained by assigning the same weight to all training examples. Then the detector is iteratively 
retrained where at each iteration more weight is given to training examples that were incorrectly classified by the 
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8. Method according to one of claims 1 to 7, wherein a basic pattern of a recordable object is stored to be detected 
in the digital representation of the photographic image to detect the location of the pattern area or image pattern. 

9. Method according to one of claims 1 to 8. wherein the pattern area represents a human face and wherein accord- 
5 ingly also the basic pattern represents a human face for instance in the shape of a pictogram. 

10. Method according to one of claims 5 to 9, wherein a colour distribution is used derived from one of said pattern 
area with the predictably known colour and/or predictably known colour distribution (both memory colour repre- 
sentations). 

10 

11. Method according to one of claims 5 to 10, wherein several distributions are provided and one distribution is se- 
lected which is deemed to match with the determined predictably known colour (memory colour). 

12. Method according to one of claims 5 to 11, wherein additional recording information is provided, providing data 
15 about light conditions, distance conditions, or the like, to provide supplemental colour correction data. 

13. Method according to one of claims 6 to 12, comprising the steps of: 

a) providing at least one set of distributions of colour values (memory colours) in the colour space, 
20 b) assigning one of said set of distributions to each of the at least one pattern areas; 

c) determine the transformation of transforming the at least one colour value of the at least one pattern area 
or image pattern such that the transformed colour value matches to the assigned distribution or distributions. 

14. Method according to one of claims 6 to 13, wherein said method being itcrativcly conducted on the basis of a 
25 respectively last colour corrected digital representation of a photographic image. 

15. Method according to one of claims 6 to 14, wherein the matching is performed in accordance with an optimisation 
process which evaluated a total matching degree between the transformed colour values and the colour values of 
the assigned distribution for each pattern area and which determines the transformation such that a function is 

30 optimised, said function mathematically combine single matching degrees for each pattern area and its assigned 

distribution. 

16. Method according to one of claims 6 to 15, wherein said distribution(s) define a probability of colour values to 
represent a replacement colour and wherein said matching degree is determined based on said probability. 

35 

17. Method according to one of claims 6 to 16, wherein the transform is determined to include a colour appearance 
transform, said colour appearance transform modelling the appearance of the colour values of the image data 
additionally by a human being, who perceives the corrected image data. 

40 18. A program which, when loaded into a computer or when running on a computer, courses the computer to perform 
a method in accordance to one of claims 1 to 17. 

19. A computer storage medium having stored a program of claim 18. 

45 20. Image processing device for processing image data, including 

a) an image data input section, 

b) an image data processing section, 

c) an image data recording section for recording image data, wherein the image data processing section is 
so embodied to implement a method to one of claims 1 to 1 9. 

21. Computer data signal embodied in a carrier wave, comprising instructions to implement a method according to 
one of claims 1 to 17. 

55 22. Data carrier device carrying control data to control a computer-based device or image processing device to im- 
plement the method according to one of claims 1 to 17. 
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Fig. 3 
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